Tests of measurement invariance failed to support the application of the "then-test"

Objective
The use of then-test (retrospective pre-test) scores has frequently been proposed as a solution to potential confounding of change scores because of response shift, as it is assumed that then-test and post-test responses are provided from the same perspective. However, this assumption has not been formally tested using robust quantitative methods. The aim of this study was to compare the psychometric performance of then-test/post-test with traditional pre-test/post-test data and assessing whether the resulting data structures support the application of the then-test for evaluations of chronic disease self-management interventions.

Study Design and Setting
Pre-test, post-test, and then-test data were collected from 314 participants of self-management courses using the Health Education Impact Questionnaire (heiQ). The derived change scores (pre-test/post-test; then-test/post-test) were examined for their psychometric performance using tests of measurement invariance.

Results
Few questionnaire items were noninvariant across pre-test/post-test, with four items identified and requiring removal to enable an unbiased comparison of factor means. In contrast, 12 items were identified and required removal in then-test/post-test data to avoid biased change score estimates.

Conclusion
Traditional pre-test/post-test data appear to be robust with little indication of response shift. In contrast, the weaker psychometric performance of then-test/post-test data suggests psychometric flaws that may be the result of implicit theory of change, social desirability, and recall bias.

Measurement invariance across chronic conditions: a systematic review and an empirical investigation of the Health Education Impact Questionnaire (heiQ™).

To examine whether lack of measurement invariance (MI) influences mean comparisons among different disease groups, this paper provides (1) a systematic review of MI in generic constructs across chronic conditions and (2) an empirical analysis of MI in the Health Education Impact Questionnaire (heiQ™).

Factor structure and measurement invariance of the Health Education Impact Questionnaire: does the subjectivity of the response perspective threaten the contextual validity of inferences?

Objective:
On-going evidence is required to support the validity of inferences about change and group differences in the
evaluation of health programs, particularly when self-report scales requiring substantial subjectivity in response generation are used as outcome measures. Following this reasoning, the aim of this study was to replicate the factor structure and investigate the measurement invariance of the latest version of the Health Education Impact Questionnaire, a widely used health program evaluation measure.
Methods:
An archived dataset of responses to the most recent version of the English-language Health Education Impact
Questionnaire that uses four rather than six response options (N=3221) was analysed using exploratory structural equation
modelling and confirmatory factor analysis appropriate for ordered categorical data. Metric and scalar invariance were
studied following recent recommendations in the literature to apply fully invariant unconditional models with minimum
constraints necessary for model identification.
Results:
The original eight-factor structure was replicated and all but one of the scales (Self Monitoring and Insight) was
found to consist of unifactorial items with reliability of ⩾0.8 and satisfactory discriminant validity. Configural, metric and scalar
invariance were established across pre-test to post-test and population sub-groups (sex, age, education, ethnic background).
Conclusion:
The results support the high level of interest in the Health Education Impact Questionnaire, particularly for use as a pre-test/post-test measure in experimental studies, other pre–post evaluation designs and system-level monitoring and evaluation.

Measurement invariance of English and French Health Education Impact Questionnaire (heiQ) empowerment scales validated for cancer

PURPOSE: If measurement invariance (MI) is demonstrated for a scale completed by respondents from two different language groups, it means that the scale measures the same construct in the same way in both groups. We assessed MI of the French- and English-language versions of the five Health Education Impact Questionnaire (heiQ) empowerment scales validated for the cancer setting. METHODS: Data came from two cross-sectional studies of Canadian cancer survivors (704 English, 520 French). Single-group confirmatory factor analysis (CFA) was used to test whether the hypothesized factor structure of the French-language heiQ empowerment scales fit the data. Multi-group CFAs were conducted to assess different levels of MI conditions (configural, metric, scalar, strict, as well as MI of factor variances, covariances, and latent means) of the French- and English-language heiQ empowerment scales. RESULTS: The correlated five-factor model showed good fit in both language groups (goodness-of-fit indices: CFI ≥ .97; RMSEA ≤ .07). Goodness-of-fit indices and tests of differences in fit between models supported MI of the five-factor model across the two language groups (∆CFI ≤ -.010 combined with ∆RMSEA ≤ .015). CONCLUSIONS: The French- and English-language heiQ empowerment scales measure the same five dimensions of empowerment in the same way across both language groups. Thus, any observed similarities or differences between French- and English-speaking respondents completing these scales are valid and reflect similarities or differences in empowerment across language groups, not measurement artifact. Consequently, heiQ empowerment data from English- and French-speaking respondents can be directly pooled or contrasted in data analyses.

Self-perceived distress and impairment in problem gamblers: a study of pre- to post-treatment measurement invariance

Gambling help services typically evaluate treatment outcomes using self-reported responses and measurements. However, gamblers’ conceptualisations and prioritisations with respect to these measurements may shift over time. Thus, changes in the self-reported responses may not always reflect true change in the individuals. This study investigated for response shift in self-report measures of psychological distress and impairment in 293 help-seeking problem gamblers. We used confirmatory factor analysis to model data structures from pre-treatment to post-treatment. The findings indicated that a response shift had occurred. Two items became less important and one item became more important in measuring psychological distress. Measurement invariance was achieved for the complete set of items for impairment. These findings provide a more in-depth understanding of the nature of self-report outcomes in otherwise routinely collected data.

Self-perceived distress and impairment in problem gamblers : a study of pre- to post-treatment measurement invariance

Gambling help services typically evaluate treatment outcomes using self-reported responses and measurements. However, gamblers’ conceptualisations and prioritisations with respect to these measurements may shift over time. Thus, changes in the self-reported responses may not always reflect true change in the individuals. This study investigated for response shift in self-report measures of psychological distress and impairment in 293 help-seeking problem gamblers. We used confirmatory factor analysis to model data structures from pre-treatment to post-treatment. The findings indicated that a response shift had occurred. Two items became less important and one item became more important in measuring psychological distress. Measurement invariance was achieved for the complete set of items for impairment. These findings provide a more in-depth understanding of the nature of self-report outcomes in otherwise routinely collected data.

Measurement of stable changes of self-management skills after rehabilitation: a latent state-trait analysis of the Health Education Impact Questionnaire (heiQ™)

To assess stable effects of self-management programs, measurement instruments should primarily capture the attributes of interest, for example, the self-management skills of the measured persons. However, measurements of psychological constructs are always influenced by both aspects of the situation (states) and aspects of the person (traits). This study tests whether the Health Education Impact Questionnaire (heiQ™), an instrument assessing a wide range of proximal outcomes of self-management programs, is primarily influenced by person factors instead of situational factors. Furthermore, measurement invariance over time, changes in traits and predictors of change for each heiQ™ scale were examined.

Measurement of stable changes of self-management skills after rehabilitation: a latent state–trait analysis of the Health Education Impact Questionnaire (heiQ™)

Methods: Subjects were N = 580 patients with rheumatism, asthma, orthopedic conditions or inflammatory bowel disease, who filled out the heiQ™ at the beginning, the end of and 3 months after a disease-specific inpatient rehabilitation program in Germany. Structural equation modeling techniques were used to estimate latent trait-change models and test for measurement invariance in each heiQ™ scale. Coefficients of consistency, occasion specificity and reliability were computed.

Gender differences in the performance of individuating acts

Although several studies in social psychology suggest that male participants are more likely than female ones to engage in individuating behaviors, other studies have found no gender differences in willingness to perform individuating acts. This study posits that differences in findings across past investigations may be attributed to the chosen domain of individuating behavior. The content of the Individuation Scale (Maslach, Stapp, & Santee, 1985) is examined in terms of Bakan's (1966) agency‐communion theory to identify two types of individuating behaviors that are consistent with men's gender role orientations (i.e., eliciting conflict, leadership), one type of individuating behavior that is consistent with women's gender role orientations (i.e., personal disclosures), and a gender‐neutral type of individuation (i.e., performance). Responses to the scale are obtained from a sample of business school students (N = 273) and a more heterogeneous mail survey sample (N = 621). A sequence of measurement invariance tests of a 4‐factor correlated model of the individuation measure indicates a high degree of equivalence in the meaning of the measure across gender groups. Subsequent latent‐means structure analysis examines gender differences in willingness to perform the 4 types of individuation behaviors captured in the scale. In the student sample, there were no mean differences in willingness to perform any of the 4 types of individuating acts. However, in the mail survey sample, findings of mean differences supported hypotheses derived from agency‐communion theory: For men as compared with women, the latent means for leadership and eliciting conflict were higher and the latent mean for personal disclosure was lower.

Measuring walking within and outside the neighborhood in Chinese elders : reliability and validity

Background : Walking is a preferred, prevalent and recommended activity for aging populations and is influenced by the neighborhood built environment. To study this influence it is necessary to differentiate whether walking occurs within or outside of the neighborhood. The Neighborhood Physical Activity Questionnaire (NPAQ) collects information on setting-specific physical activity, including walking, inside and outside one's neighborhood. While the NPAQ has shown to be a reliable measure in adults, its reliability in older adults is unknown. Additionally its validity and the influence of type of neighborhood on reliability and validity have yet to be explored. Methods : The NPAQ walking component was adapted for Chinese speaking elders (NWQ-CS). Ninety-six Chinese elders, stratified by social economic status and neighborhood walkability, wore an accelerometer and completed a log of walks for 7 days. Following the collection of valid data the NWQ-CS was interviewer-administered. Fourteen to 20 days (average of 17 days) later the NWQ-CS was re-administered. Test-retest reliability and validity of the NWQ-CS were assessed. Results : Reliability and validity estimates did not differ with type of neighborhood. NWQ-CS measures of walking showed moderate to excellent reliability. Reliability was generally higher for estimates of weekly frequency than minutes of walking. Total weekly minutes of walking were moderately related to all accelerometry measures. Moderate-to-strong associations were found between the NWQ-CS and log-of-walks variables. The NWQ-CS yielded statistically significantly lower mean values of total walking, weekly minutes of walking for transportation and weekly frequency of walking for transportation outside the neighborhood than the log-of-walks. Conclusions : The NWQ-CS showed measurement invariance across types of neighborhoods. It is a valid measure of walking for recreation and frequency of walking for transport. However, it may systematically underestimate the duration of walking for transport in samples that engage in high levels of this type of walking.

Body image during pregnancy: an evaluation of the suitability of the body attitudes questionnaire

Background: 

Cross-cultural adaption of the Health Educaiton Impact Questionnaire: experimental study showed expert committee, not back-translation, added value

Objectives
To assess the contribution of back-translation and expert committee to the content and psychometric properties of a translated multidimensional questionnaire.

Study Design and Setting
Recommendations for questionnaire translation include back-translation and expert committee, but their contribution to measurement properties is unknown. Four English to French translations of the Health Education Impact Questionnaire were generated with and without committee or back-translation. Face validity, acceptability, and structural properties were compared after random assignment to people with rheumatoid arthritis (N = 1,168), chronic renal failure (N = 2,368), and diabetes (N = 538). For face validity, 15 bilingual people compared translations quality with the original. Psychometric properties were examined using confirmatory factor analysis (metric and scalar invariance) and item response theory.

Results
Qualitatively, there were five types of translation errors: style, intensity, frequency/time frame, breadth, and meaning. Bilingual assessors ranked best the translations with committee (P = 0.0026). All translations had good structural properties (root mean square error of approximation <0.05; comparative fit index [CFI], ≥0.899; and Tucker–Lewis index, ≥0.889). Full measurement invariance was observed between translations (ΔCFI ≤ 0.01) with metric invariance between translations and original (lowest ΔCFI = 0.022 between fully constrained models and models with free intercepts). Item characteristic curve analyses revealed no significant differences.

Conclusion
This is the first experimental evidence that back-translation has moderate impact, whereas expert committee helps to ensure accurate content.

A Cross-Cultural Evaluation of the Personal Wellbeing Index – School Children in Samples of Australian and Portuguese Adolescents

The practice of comparing nations on subjective wellbeing (SWB) is becoming commonplace, with many countries ranked by economists and social scientists alike according to average levels of SWB based on survey responses. Such large, multi-national population surveys have the potential to generate insights into the causes and correlates of SWB within different cultural groups, as well as inform policy regarding how to improve the wellbeing of citizens. At the heart of these large-scale research endeavors are SWB measures that function equivalently between the various participating cultural groups. For this reason, it is concerning that their remains a paucity of research that supports measurement equivalence for many SWB instruments commonly employed. Thus, it remains unclear whether variations in SWB across cultures reflect true differences, or whether these differences reflect measurement biases (e.g., response bias inherent within a particular cultural group). The aim of this study was to examine the psychometric equivalence of the Personal Wellbeing Index–School Children (PWI-SC) in convenience samples of Australian and Portuguese adolescents using multiple-group confirmatory factor analysis. Participants comprising the Australian sample were 1104 Victorian high-school students aged between 12 and 19 years (M = 14.42, SD = 1.63). Participants comprising the Portuguese sample were 573 high-school students living in Portugal aged between 12 and 18 years (M = 14.32, SD = 1.72). The results demonstrated strict factorial invariance between both versions of the PWI-SC, suggesting that this scale measures the same underlying construct in both samples. Moreover, these findings provide preliminary support for quantitative comparisons between Australian and Portuguese adolescents on the SWB variable as valid.

On the validity of measuring change over time in routine clinical assessment: a close examination of item-level response shifts in psychosomatic inpatients

Objective: Significant life events such as severe health status changes or intensive medical treatment often trigger response shifts in individuals that may hamper the comparison of measurements over time. Drawing from the Oort model, this study aims at detecting response shift at the item level in psychosomatic inpatients and evaluating its impact on the validity of comparing repeated measurements. Study design and setting: Complete pretest and posttest data were available from 1188 patients who had filled out the ICD-10 Symptom Rating (ISR) scale at admission and discharge, on average 24 days after intake. Reconceptualization, reprioritization, and recalibration response shifts were explored applying tests of measurement invariance. In the item-level approach, all model parameters were constrained to be equal between pretest and posttest. If non-invariance was detected, these were linked to the different types of response shift. Results: When constraining across-occasion model parameters, model fit worsened as indicated by a significant Satorra–Bentler Chi-square difference test suggesting potential presence of response shifts. A close examination revealed presence of two types of response shift, i.e., (non)uniform recalibration and both higher- and lower-level reconceptualization response shifts leading to four model adjustments. Conclusions: Our analyses suggest that psychosomatic inpatients experienced some response shifts during their hospital stay. According to the hierarchy of measurement invariance, however, only one of the detected non-invariances is critical for unbiased mean comparisons over time, which did not have a substantial impact on estimating change. Hence, the use of the ISR can be recommended for outcomes assessment in clinical routine, as change score estimates do not seem hampered by response shift effects.

Meaning in life in earlier and later older-adulthood : confirmatory factor analysis and correlates of the meaning in life questionnaire

OBJECTIVE: To validate the Meaning in Life Questionnaire (MLQ) in earlier and later older-adulthood, and examine its correlates. METHOD: Participants in earlier (n = 341, M age = 68.5) and later older-adulthood (n = 341, M age = 78.6) completed the MLQ and other measures. Confirmatory multigroup analysis, correlations, and regression models were conducted. RESULTS: A two-factor (presence and search), eight-item model of the MLQ had a good fit and was age-invariant. Presence and search for meaning were largely unrelated. Meaning was associated with life satisfaction, well-being across a range of domains, and psychological resources. Searching for meaning correlated negatively with these variables, but to a lesser degree in later older-adulthood. DISCUSSION: The MLQ is valid in older-adulthood. Meaning in life is psychologically adaptive in older-adulthood. Searching for meaning appears less important, especially in later older-adulthood. Findings are discussed in the context of aging and psychosocial development.